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ABSTRACT 

The two conference papers comprising this document 
address course development and assessment procedures used in distance 
education at the West German Open University. The first paper , on 
course design and development techniques used in distance teaching , 
reviews some of the techniques of course evaluation ^nd course design 
that have been used in the Open University and emphasizes the 
importance of the relationship between the two. It starts by 
discussing methods used in the Open University's Science Foundation 
Course to strengthen this relationship, and concludes with a specific 
example of how a particular design prescription might be modified in 
the light of information collected. The second paper, on assessment 
procedures used in distance teaching, reviews some of the procedures 
that have proved to be of value in developing assessment material, 
primarily for grading purposes. The paper begins by stressing the 
need to identify the purpose behind any form of assessment used, and 
discusses the type of anomalies that can arise if this is not done. 
It then addresses the role that item analysis and behavioral 
objectives plc*y in the development of assessment material. Each paper 
includes references. (GL) 
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1 Introduction 



This paper reviews some of the techniques of 
course evaluation and course design that have 
been used in the Open University, and emphasises 
the importance of the relationship between the 
two. It starts by looking at methods used in the 
Open University's Science Foundation Course to 
strengthen this relationship, and concludes with 
a specific example of how a particular design 
prescription might be modified in the light of 
information collected. 
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Course Evaluation and Related 
Decision Making 



To understand the methods used on the Science 
Foundation Course we must first take a brief 
look at the development cycle for that course. 
It was produced over a period of two years by 
a team consisting of 15 subject specialists, 
2 editors, a number of BBC producers, an edu- 
cational technologist and several consultants. 
Work began in 1969, and the course was presented 
to the student population in 19 M after going 
through a number of drafts. The course is 
currently being remade, and the new course 
will be presented to students in 1979. 

Diagram 1 . Course Development Cycle of Science 
Foundation Course 

PRODUCTION PERIOD OF FIRST PRESENTATION 

i i i i i i i i i i I 

1969 70 71 72 73 74 75 76 77 78 79 
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SECOND 
PRESENTATION 



I I I 



1976 77 78 79 80 81 82 



Evaluation of the course began long before it was 
first presented to students, and has continued to 
the present date. The techniques used are de- 
scribed in some detail in "Course Evaluation at 
the Open University, A Case Study * 1 ' " , and are 
summarised in the table below. 



Table 1 . Course Evaluation Techniques On The 
Science Foundation Course 



- Jw ; Production 
Feedback ^ i 

— . -w. 


Presentation 


Course Teairi C.T. Discussions 

Papers 




Students j Interviews 

! De vo 1 opme r 1 1 a 1 
! Testing 


Interviews 
Surveys 
Curf 

CMA Feedback 
Student Letters 
Item Analysis 


Regional 
Staff 


C.r. Djscuefloiis 


Surveys (CT4) 
Tutor Notes (TMAs) 
Counsellor Notes 

(CMAs) 


Research 
Groups 


Reports 

Aids to Design 
and Decision Making 


Repor ts 

Aids to Design 
and Decision Making 



By 19 75 so much information had been gathered on 
the course that it had become clear that the main 
problem was not how to evaluate the course, but 
how to ensure that all the information collected 
was effectively used. Emphasising this need was 
the knowledge that the most substantial modifi- 
cations to the course had been made as it went 
through successive drafts in the early stages of 
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production, at a time when the feedback infor- 
mation available was both limited , and subjective, 
in nature. Once the course, with all its inter- 
related components, had been presented to students 
economic considerations alone limited the modi- 
fications that were possible. The situation was 
therefore somewhat anomalous in that the 
quantity, and quality, of information available 
to inform decision making had increased con- 
siderably, bat the opportunities for modifying 
the course were much more limited than during 
the period of course production. 

The solution envisaged was to focus attention on 
the remake of the course when major changes would 
be possible. The idea was to identify, as far as 
possible, questions of principle that would be 
discussed during the early phases of the remake, 
and to make sure that all possible information 
was made available to course team members in order 
to ensure that decisions werde made on the most 
informed basis possible. With so much information 
available an information bank, and a rapid re- 
trieval system, was seen as essential. 

The Bank was set up during 1976 and contains a 
wide variety of information not only on the 
course itself but also from a number of other 
relevant sources both inside and outside the 
university. The information gathered varies in 
nature from discussion papers to highly objective 
statistical reports. The main intent of those 
developing the bank is to ensure that, when 
specific questions are debated during the early 
phases of the remake of the course, that relevant 
information will be available in a readily 
digested form to help inform decision making. 



Information is thus brought to bear at a time 
when change is most likely to occur. The process 
envisaged is summarised in diagram 2, 



Diagram 2. System of ensuring informed decision making with help 
of information bank. 
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The philosophy involved is an extremely simple 
one. Design techniques are most likely to be 
improved if course teams are made aware of the 
deficiencies of existing techniques at a time 
when change is actually possible. Evaluation is 
thus seen as a means of creating an awareness of 
a need. Nor should the work of development cease 
at this point. Evaluative information may be 
seen as threatening or unhelpful if presented 
alone, and it is important that it should be 
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supported by positive prescriptions for improve- 
ment. 

3 Prescriptions for Course Design 

In behavioural objectives we have an interesting 
example not only of a tool used in course design, 
but also of how prescriptions for course design 
might be improved in the light of information 
collected from a variety of sources. 

Behavioural objectives have been used in a 
variety of forms in Open University courses, and 
the Science Foundation Course provides us with 
an interesting example with behavioural objectives 
stated in detail at the beginning of each Unit. 
But we might ask whether student learning might 
be more enhanced if behavioural objectives are 
placed at the end of each Unit, or spread through- 
out each Unit so that each objective is close to 
the related contpnt. Research on behavioural 
objectives concerning this question is very 
limited, but considerable insights can be gained 
by looking at related research concerning the 
use of questions inserted into texts. 

A number of such studies are reviewed in the 

paper entitled "Resolution of Conflicting Claims 

(2) 

Concerning Behavioural Objectives' . The 
studies reviewed were concerned with the effect 
on student learning of inserting questions into 
texts either immediately before, or immediately 
after, the passages to which they referred. The 
question-related passages were described as 
relevant information and the questions as pre-, 
or post-, questions according to their location. 



In reading the instructional material containing 
either pre-, or post-, questions, students were 
not permitted to review any section of the text 
once it had been read. On completion they were 
tested to determine not only their acquired 
knowledge of relevant information, but also that 
of incidental information (that is textual infor- 
mation not related to the inserted questions) . 

A number of findings emerged from the studies. 
First, relevant learning was enhanced by the 
provision of inserted questions, with post- 
questions more effective than pre-questions in 
this respect. Second, incidental learning was 
enhanced by the provision of post-questions, but 
not by pre-questions which in some cases de- 
pressed incidental learning. Finally, it was 
oberserved that the above effects were more 
likely to be observed when questions were in- 
serted one at a time into the text (before, or 
after, every 10 sentences for example) than when 
the same number of questions were inserted into 
the text in groups (for example, 5 questions at 
a time before, or after, every 50 sentences) . 

From the studies it would appear that the same 
questions function in different ways according 
to their location in the text. Placed immediately 
before related passages they appear to function 
as orienting stimuli "directing student attention 
to the relevant and away from the incidental. 
The effect is to enhance relevant learning but 
to depress incidental learning. Questions inser- 
ted immediately after related passages, however, 
appear to function as reinforcement stimuli, 
stimulating further consideration of the relevant 

without depressing incidental learning that has 
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already taken place. The fact that post- 
questions can enhance not only relevant 
learning but also incidental learning suggests 
that when this occurs the two types of 
information are related. 

Since behavioural objectives should clearly 
indicate what a student should be able to do 
as a result of the learning process, and how 
what he does should be measured, one might 
expect behavioural objectives to function in 
much the same way as inserted questions, pro- 
ducing orienting, or reinforcement, stimuli 
according to their location in the text, or 
according to the way in which they are used. 
With the research on inserted questions in mind 
it would appear logical to recommend that in 
future behavioural objectives should not be 
placed together at the beginning or end of 
each Unit, but should be inserted separately 
at frequent intervals into the text immediately 
after each related passage. Such objectives 
would be more likely to function as reinforce- 
ment stimuli than objectives placed immediately 
before each related passage. One might 
hypothesise that such post-objectives should 
enhance relevant learning more than pre- 
objectives, and should be less likely to 
depress incidental learning. 



Needless to say behavioural objectives are used 
for a variety of purposes, and it is clear that 
the above discussion is limited to the use of 
behavioural objectives to enhance student 
learning. Other uses of behavioural objectives 
are discussed in the related paper on 
"Assessment Procedures Used in Distance 
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1 Introduction 



This paper is a review of some of the procedures 
that have proved to be of value in developing 
assessment material, primarily for grading 
purposes, at the Open University. The paper 
begins by stressing the need to identify the 
purpose behind any form of assessment used, and 
discusses the type of anomalies that can arise 
if this is not done. It then goes on to look 
at the part which item analysis and behavioural 
objectives may play in the development of 
assessment material. 

2 Purposes of Assessment 

Assessment material might be developed for a 
variety of reasons, and it is important to keep 
the relevant purpose of any assessment clearly 
in mind. For example, it may be used 

To grade students 

To measure student mastery of objectives 
To aid student learning. 

Any confusion of purpose may lead to particular 
problems. For example, one cannot expect tests 
designed to measure student mastery of 
objectives to be appropriate for grading 
purpose, for if the majority of students master 
the majority of test items the test will not be 
an accurate means of placing students in rank 
order. 

In the past assessment material at the Open 
University has been designed with particular 
standards (mastery) in mind, but at the same 
time it has been used for grading purposes. 
This gives rise to anomalies particularly where 
scores are to be added. The following simple 



examples should Illustrate the point. 

Table 1 indicates hypothetical raw scores for a 
group of 5 students on two separate tests , one 
concerned with biology and one with physics. 
Since the students are to be ranked in order 
of ability in science the two scores , which 
place students in opposite rank order, are to 
be added together. If the raw scores are added 
together, as in table 1, it will be noted that 
the final overall order is determined by the 
biology test, that is the test with the largest 
standard deviation of scores. 

Table 1 



Test 
Student*^^ 


Biology 


Physics 


Total 


A 


70 


(1) 


18 


(5) 


88 (1) 


B 


60 


(2) 


27 


(4) 


87 (2) 


C 


50 


(3) 


36 


(3) 


86 (3) 


D 


40 


(4) 


45 


(2) 


85 (4) 


E 


30 


<*) 


54 


(1) 


84 (5) 


Mean (M) 


50 




36 






Standard ( s) 
Deviation 






yu2 







Addition of Raw Scores (X) with different means 
and different standard deviations. 

If the physics scores in table 1 are multiplied 
by 10/9 the standard deviations for both test 
scores will be the same, and the two tests will 
contribute equally to the final rank order 
(table 2) . 



Table 2 



"^^ITest 


Bio? ogy 


Physics 


Total 


IV 

A 


70 


(D 


20 


(5) 


90 (1=) 


B 


60 


(2) 


30 


(4) 


90 (1=) 


C 


50 


(3) 


40 


(3) 


90 (1=) 


D 


40 


(4) 


50 


(2) 


90 (1=) 


E 


30 


(5) 


60 


(D 


90 (1=) 


Mean (M) 


50 




40 






Standard (s) 
Deviation 


yioo 




J 200 







Addition of Adjusted Scores with different means 
but same standard deviations. 



Altering the mean scores has no effect on the 
overall rank order as can be see** from table 3. 
However, for purposes of comparing performances 
on tests it is useful to adjust tests to have 
the same mean as well as the same standard 
deviation. 

Table 3 





Biology 


Physics 


Total 


A 


70 


(1) 


30 


(5) 


100 (1=) 


B 


60 


(2) 


40 


(4) 


100 (1=) 


C 


50 


(3) 


50 


(3) 


100 (1=) 


D 


40 


(4) 


60 


(2) 


100 (1=) 


E 


30 


(5) 


70 


(1) 


100 (1=) 


Mean (M) 

Standard (s) 
Deviation 


50 
^200 


50 
J200 





Addition of Standard Scores with same means 
and same standard deviations. 



The standardisation of scores is not sufficient 
in itself to remove all grading anomalies when 
scores are added, for standardisation alters the 
means and standard deviations but not the shape 
of the distribution concerned. Thus we might 
have two students, A and B, with what appears 
to be roughly comparable performances on the 
two tests. Student A is the top of the class 
in physics but the bottom in biology. Stucnt B 
is the bottom in physics but the top in biology 
(Graphs 1 and 2) . Because of the sktwness of 
the biology scores student B is ranked top on 
the combined scores. If the test scores are 
normalised this anomaly will disappear. 

Graph "J 




If scores are not normalised prior to addition 
for grading purposes corrse teams need to answer 
a number of difficult questions. A few of these 
are raised with reference to data (table 4) 



collected from 16 computer marked assignments 
presented to students on the Science Foundation 
Course in 1974. 



Table 4. Data Taken From Computer Marked Assign 
ments Presented to Students on Science 
Foundation Course in 1974. 



^"""■-^^ssignnent 
Index^<^^ 


41 


42 


43 


44 


45 


46 


47 


48 


Mean Score 

Standard 
Deviation 


85.2 
13.0 


77.3 
13.2 


78.0 
15.0 


78.0 
13.2 


77.4 
14.6 


66.6 
16.0 


64.2 
15.7 


72.5 
19.1 




^"""^•^ssignment 
Index 


49 


50 


51 


52 


53 


54 


55 


56 


Mean Score 

Standard 
Deviation 


80.9 
13.0 


71.6 
12.0 


68.4 
14.2 


62.2 
11.0 


51.5 
9.2 


81.2 
11.0 


82.9 
11.6 


55.6 
9.7 



The following are two such questions which course 
teams would need to answer satisfactorily if 
they were to object to the assignment scores 
being normalised prior to addition for grading 
purposes . 

Should assignment 48 contribute more 
to a student's overall score than 
assignment 53? The differences in 
standard deviations will certainly 
have this effect in determining a 
student's overall ranking. 



If a student achieves a score of 85.2 
on assignment 41 but only 51.5 on 
assignment 53 does it follow that his 
level of performance has fallen badly? 
Does the difference in mean scores 
indicate differing student abilities 
in the two areas or different standards 
of test design? 



3 Item Analysis 

Where tests have been designed primarily to 
grade students test items and tests may be 
improved with the help of item analysis. Such 
a facility is used in the Open University pro- 
viding a range of statistics on both individual 
items and tests as a whole. The nature of the 
statistics used are fully described in "Item 
Analysis at the Open University: A Case Study * 1 
Two statistics might be picked out for further 
comment. The Discrimination Index is particular- 
ly important since it tells us to what extent 
each item helps to rank students on the test as 
a whole , or in other words to what extent each 
item discriminates between students. The 
Coefficient of Internal Consistency is a 
further statistic of particular value which 
provides us with an indication of the re- 
liability of the test as a whole and hence 
with a measure of the error associated with 
any score awarded on the test. The reliability 
of a test may normally be increased by in- 
creasing the number of items with good 
discrimination in the test. 



Item analysis has two distinct roles to play 



in student assessment. In the first instance, it 
may be used to help identify gross anomalies in 
items, enabling such items to be removed from a 
test before scores are actually awarded to 
students. In the second instance it may be used 
to build up banks of items with known character- 
istics. Such items may be used to produce 
reliable tests for future use. The Procedures 
envisaged are summarised in diagram 1. 



Diagram 1 . Method of Using Item 
Analysis Data. 



instructional oateriel 
It ptudonta 



P rooon t ralatao taat 
to atudanta 



ISutjoi 
Lwrt i 



Ject provisional taat 
a ta Itaaj Analyuia 



ono—li— 



/ Da annual i*a axiat in A 
I indicia of itaai H 
V reviewed? / 



No 



\> Yoa 



Do not pomit itoa to 
count toverda atudent 
oceree on toot if 

small !■ mi mn 



i 



Accept r a la tad 


ino traction 


•1 ■eteriaU 


for aubaeoju 


•nt tenchiaaa 


froajrojaoaa 




\ 


f 


Ubel taat 


it a* with 


appropriata 


indlcee 


ond placa in Itaai 


lank for ou 
ueaoe 


aeeepaant 



Study toot itaai and 
rolatad inotructionol 
amtorial . 



1 



Redraft taat ito 
ond euknit to 
furthor triate 



Yoa 



(la it 
rawed 
rpdraj 



± 



poaaibla to 
fdy anoawly by 
redraft ino taat it—? 



ftodra/t inotruc- 
tionol notarial 
ond eubnit to 
furthor triolo 



Yaa 



Xa it poaaibla to 
ronody anowaly by / 
redrafting tha 
inotructionol notarial?! 



^ No 



•eject toot ltena 
anf. inatructionol 
notarial that cannot 
00 oultablr radraftad 



ERLC 



2. 



4 Behavioural Objectives 



The paper on "Behavioural Objectives" 1 ' 
describes the nature of behavioural objectives 
and provides basic rules concerning their 
construction , while the paper on "Resolution 
of Conflicting Claims Concerning Behavioural 
Objectives"* ' reviews research into how student 
learning is affected by the provision of 
behavioural objectives. Here our particular 
interest is limited to how such objectives 
might be used to help develop assessment 
material. In the process it is inevitable that 
we will also see how it can be used to help 
structure and analyse instructional material. 



The procedure recommended is one of deriving 
detailed objectives step by step from general 
statements of educational intent, recording 
each step in the form of a flow diagram. This 
does not imply that only one set of behavioural 
objectives can be derived from the initial 
statements. Nothing is further from the truth. 
However , what it does is to make explicit the 
logic used in deriving stated objectives thus 
£ opening up that logic to public inspection and 
discussion. Once detailed objectives have been 
stated the type of assessment required to 
measure these objectives becomes much clearer. 



A flow diagram (diagram 2) used in the actual 
development of a physics Unit should illustrate 
the point. The author's ultimate aim was that 
students should be able to: 

Explain how the performance of complete 
imaging systems utilising any form of 
radiation may be described in terms of 
transfer functions. 
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Diagram 2. Derivation of more Specific 
Objectives from a General 
Learning Outcome. 
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Working backwards from this the author believed 
that this might be achieved if students were 
able to explain the performance of a number of 
imaging systems, such as the microscope, in 
terms of transfer functions. Each sub-aim could 
be similarly broken down into more specific 
terms. Thus the author felt wuat if students 
were to be able to explain the performance of 
the microscope in terms of transfer functions 
they would need to explain the performance of 
the individual microscope parts in similar 
terms. 



An author may use such logic to help him 
structure his instructional material while 
course team members may use the flow diagrams 
developed to review weaknesses in the structure. 
In addition authors may be encouraged to produce 
as many test items as possible for each of the 
most specific objectives identified. Each 
objective might then be stated in very specific 
terms by stating that it will be measured by 
means of a representative sample of items 
selected at random from those related to the 
specified objective. If an item bank is used 
for this purpose an equivalent sample of items 
may be made available to students and future 
test designers , the items related to a given 
objective being identified by such character- 
istics as indices of facility and discrimination 
as well as indicators of content and objective 
covered. 

Needless to say this paper cannot be more than 
a brief review of a limited number of aspects 
of assessment. Those wishing to pursue the 
topic further may find it useful to look at a 
series of papers related to the Open Univer- 
sity's Computer Marked Assignment System. 
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